I am guessing we do not have enough xfer registers to indirect_ref xdp_meta_base read this in a block?
This must be the case since this macro is called from a block of code that calls pv_invalidate_cache

Yes, this is the case. What is the right way to fix this?

I think you can to re-use $__pv_pkt_data[32]. This is used to cache packet header data. When you use pv_save_xdp_meta in ebf_reentry, you can do pv_invalidate_cache(in_vec) afterward. On the next pv_seek the packet header will be re-read.

These changes will clash with other work Netronome has incoming. Specifically, we want to grow the Seq Ctx field by one bit, and we have a different use for the X bit. We might have to add another word to the PV, which has a knock-on effect everywhere else. @mrapson-netronome and I are discussing this.

Sure, I was ready to re-implement the layout when pushing the patch. For me ideal would be to have at least 4+ bits to store the length of XDP meta (in words).

XDP meta is now <= 32 bytes and in 4 bytes chunks, so I need at least 4 bits to store the length. Maybe in future it will be extended to 64 or 128 bytes (still in 4 bytes chunks though), so I would like to have 6+ bits, so that we won't have to change firmware/kernel ABI for this change.

The situation is that we are running out of local memory on the datapath ME's:
This is the layout without your change:

512                 i32.me0._PKT_IO_PKT_VEC
256                 i32.me0.gro_cli_lm_ctx
32                  i32.me0.__actions_sriov_keys
16                  i32.me0.nfd_in_ring_info
16                  i32.me0.nfd_out_ring_info
2048                i32.me0.EBPF_STACK_BASE
Total used 2880 of 4096B

With your change:

1024                i32.me0._PKT_IO_PKT_VEC
256                 i32.me0.gro_cli_lm_ctx
32                  i32.me0.__actions_sriov_keys
16                  i32.me0.nfd_in_ring_info
16                  i32.me0.nfd_out_ring_info
2048                i32.me0.EBPF_STACK_BASE
Total used 3392B of 4096B

You will notice that changing PV_SIZE_LW from 16 to 24 doubles the _PKT_IO_PKT_VEC size. That is because it must be a power-of-2 size. See the alloc logic in pv.uc line 304. The upshot is that there is 8 more words available in the PV that is not shown in the ASCII art. Perhaps you can move your extra fields to this unused section?

If we add to the above the features that Netronome plans to introduce soon:

1024                i32.me0._PKT_IO_PKT_VEC
640                 i32.me0.gro_cli_lm_ctx
16                  i32.me0.nfd_in_ring_info
16                  i32.me0.nfd_out_ring_info
32                  i32.me0._PKT_IO_RX_META
256                 i32.me0.__actions_keys <- had to "break" alignment
2048                i32.me0.EBPF_STACK_BASE
Total used 4032B of 4096B

You can see that we are really out of LM, (and I have to violate the alignment requirement of symbol __actions_keys). Thus there really isn't space to grow the XDP metadata.

If we need to go above 32B, we'll have to consider alternatives, like maintaining the metadata in CLS, or directly in the packet buffer.

Do we need to lock down LM addr2 for this? Are we sure it's not used elsewhere?

Edit: I think you have to update pv_push and pv_pop as well?

I am allocating data by extending PV_SIZE_LW to 24 words. Shouldn't pv_push and pv_pop work as is?

Maybe... It certainly looks like it from inspecting the code. (Note to self, write a unit test to confirm this)